LARGE PRETRAINED MODELS

Papers with large pretrained models

Best Practices for Data-Efficient Modeling in NLG:How to Train Production-Ready Neural Models with Less Data (2020.coling-industry)

Copied to clipboard

Ankit Arun, Soumya Batra, Vikas Bhardwaj, Ashwini Challa, Pinar Donmez, Peyman Heidari, Hakan Inan, Shashank Jain, Anuj Kumar, Shawn Mei, Karthik Mohan, Michael White

Natural Language Interfaces

Challenge:	Natural language generation (NLG) is a critical component in conversational systems . Traditionally, NLG components have been deployed using template-based solutions . however, deployment of such model-based systems has been challenging due to high latency and data needs.
Approach:	They propose a family of techniques to deploy data-efficient neural solutions for NLG in conversational systems to production.
Outcome:	The proposed techniques achieve production quality with light-weight neural network models using fraction of the data needed otherwise.

On the use of BERT for Neural Machine Translation (D19-56)

Copied to clipboard

Stephane Clinchant, Kweon Woo Jung, Vassilina Nikoulina

Multilinguality

Challenge:	Existing studies on using pretrained language models for supervised NMT have not been successful.
Approach:	They propose to integrate BERT pretrained models with supervised NMT models by using monolingual data.
Outcome:	The proposed models improve translation quality in English-German, English-Russian and IWSLT14 datasets.

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)

Copied to clipboard

Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei

Language Models

Challenge:	Large pre-trained models suffer from domain shift and are not optimal for specific domains.
Approach:	They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains.
Outcome:	The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE.

Disentangling Representations of Text by Masking Transformers (2021.emnlp-main)

Copied to clipboard

Xiongyi Zhang, Jan-Willem van de Meent, Byron Wallace

Semantic Text Processing

Challenge:	Large pretrained models such as BERT encode a range of features into monolithic vectors, providing strong predictive accuracy across downstream tasks.
Approach:	They explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects.
Outcome:	The proposed method disentangles sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics.

Comparing Test Sets with Item Response Theory (2021.acl-long)

Copied to clipboard

Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman

Linguistic Theories

Challenge:	Recent results from large pretrained models show that many datasets are saturated and unlikely to detect further progress.
Approach:	They evaluate 29 datasets using predictions from 18 pretrained Transformer models on individual test examples.
Outcome:	The proposed datasets are saturated and unlikely to detect future improvements.

Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison (2025.naacl-long)

Copied to clipboard

Tsz Kin Lam, Marco Gaido, Sara Papi, Luisa Bentivogli, Barry Haddow

Multimodality

Challenge:	Large Language Models (LLMs) have been successful in NLP tasks, but there is growing interest in extending their capabilities to speech.
Approach:	They propose to use dense feature prepending (DFP) to integrate speech into LLMs to enable end-to-end training with a speech encoder.
Outcome:	The proposed approach does not show a clear advantage over cross-attention.

AutoLoRA: Automatically Tuning Matrix Ranks in Low-Rank Adaptation Based on Meta Learning (2024.naacl-long)

Copied to clipboard

Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, Pengtao Xie

Language Models

Challenge:	Large-scale pretraining followed by task-specific finetuning has achieved great success in various NLP tasks.
Approach:	They propose a meta learning based framework for automatically identifying the optimal rank of each LoRA layer.
Outcome:	The proposed framework is based on a meta learning based framework that can identify the optimal rank of each LoRA layer.

ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference (2022.findings-acl)

Copied to clipboard

Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ji Ma, Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, Donald Metzler

Language Models

Challenge:	State-of-the-art neural models typically encode document-query pairs using cross-attention for re-ranking.
Approach:	They propose to fine tune a pretrained encoder-decoder model using document to query generation.
Outcome:	The proposed model achieves comparable results to more expensive approaches while being 6.8X faster.

Efficient Shapley Values Estimation by Amortization for Text Classification (2023.acl-long)

Copied to clipboard

Chenghao Yang, Fan Yin, He He, Kai-Wei Chang, Xiaofei Ma, Bing Xiang

Information Retrieval

Challenge:	Shapley Values are often estimated with a small number of stochastic model evaluations, but this can only be mitigated by aggregating thousands of model evaluation.
Approach:	They propose to combine a model with thousands of model evaluations to estimate Shapley Values without additional model evaluation.
Outcome:	The proposed model estimates Shapley Values accurately with up to 60 times speedup compared to traditional methods and does not suffer from stability issues as inference is deterministic.

Response Selection for Multi-Party Conversations with Dynamic Topic Tracking (2020.emnlp-main)

Copied to clipboard

Weishi Wang, Steven C.H. Hoi, Shafiq Joty

Language Models

Challenge:	Existing response selection methods focus on a two-party single-conversation scenario.
Approach:	They propose a multi-task learning framework that frames response selection as a dynamic topic tracking task to match the topic between the response and relevant conversation context.
Outcome:	The proposed framework outperforms existing methods on an Ubuntu IRC dataset in response selection and topic disentanglement tasks.

Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator (2023.findings-acl)

Copied to clipboard

Ziwei He, Meng Yang, Minwei Feng, Jingcheng Yin, Xinbing Wang, Jingwen Leng, Zhouhan Lin

Language Models

Challenge:	Existing transformer models are computationally demanding and prohibitively costly for long sequences due to the quadratic complexity of its selfattention module.
Approach:	They propose a transformer-based model that inherits weights from large pretrained models by removing redundancies in hidden sequences using the ready-made Fast Fourier Transform operator.
Outcome:	The proposed model outperforms the standard BART model on the long-range modeling benchmark LRA with significant improvements in speed and space.

Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)

Copied to clipboard

Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, Fei Huang

Language Models

Challenge:	Recent pretrained language models extend from millions to billions of parameters.
Approach:	They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process.
Outcome:	The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins.